Reversal Strategies for Adjoint Algorithms

Adjoint Algorithms are a powerful way to obtain the gradients that are needed in Scientific Computing. Automatic Differentiation can build Adjoint Algorithms automatically by source transformation of the direct algorithm. The specific structure of Adjoint Algorithms strongly relies on reversal of the sequence of computations made by the direct algorithm. This reversal problem is at the same time difficult and interesting. This paper makes a survey of the reversal strategies employed in recent tools and describes some of the more abstract formalizations used to justify these strategies. 1 Why build Adjoint Algorithms? Gradients are a powerful tool for mathematical optimization. The Newton method for example uses the gradient to find a zero of a function, iteratively, with an excellent accuracy that grows quadratically with the number of iterations. In the context of optimization, the optimum is a zero of the gradient itself, and therefore the Newton method needs second derivatives in addition to the gradient. In Scientific Computing the most popular optimization methods, such as BFGS [16], all give best performances when provided gradients too. In real-life engineering, the systems that must be simulated are complex: even when they are modeled by classical mathematical equations, analytic resolution is totally out of reach. Thus, the equations must be discretized on the simulation domain, and then solved e.g. iteratively by a computer algorithm.